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EXAMINING  THE  EFFECT  OF  INFORMATION  SEQUENCE  ON  EXPERT  JUDGMENT 


INTRODUCTION 


It  is  clear  that  military  decision  makers  will  be  working  increasingly 
with  computerized  systems  intended  to  aid  their  cognitive  processes  and  reduce 
their  cognitive  workload.  What  is  less  clear  is  whether  these  systems  can  be 
based  on  a  growing  body  of  cognitive  research  findings,  and  on  the  integration 
of  these  findings  into  theories  that  provide  general  design  principles. 

For  example,  there  is  substantial  basic  research  (e.g.,  see  the  anthology 
by  Kahneman,  et  al.,  1982)  demonstrating  that  people  use  heuristics  (i.e., 
procedures  or  rules)  to  make  many  judgments  and  decisions.  Moreover,  the 
research  indicates  that  these  heuristics  are  in  large  part  determined  by  the 
judgment  task  facing  the  person.  For  many  tasks,  these  heuristics  can  result 
in  systematic  biases  or  errors  in  judgment.  An  important  question  given 
minimal  investigation  thus  far  is  whether  military  personnel  use  specific 
heuristics  when  performing  tasks  employing  computer  systems  and,  if  so, 
whether  these  heuristics  result  in  systematic  biases  that  can  affect  perfor¬ 
mance  of  these  tasks. 

This  paper  presents  the  results  of  an  experiment  investigating  whether 
operators  for  the  Army's  Patriot  Air  Defense  System  employ  an  anchoring  and 
adjustment  heuristic  when  making  "friend  versus  foe”  identification  judgments. 
If  they  do,  then  the  belief  updating  model  proposed  by  Einhorn  and  Hogarth 
(1987)  and  Hogarth  and  Einhorn  (1989)  predicts  that  operators'  judgments  will 
result  in  an  "information  order  bias"  under  certain  conditions  and  not  others. 
In  particular,  their  model  predicts  that  when  there  is  only  one  aircraft 
track  on  the  Patriot  display  at  a  time  and  there  is  conflicting  information 
about  the  aircraft  --  a  situation  representative  of  that  when  the  U.S.S. 
Vincennes  shot  down  an  Iranian  airliner  --  the  same  information  will  result  in 
different  judgments  simply  depending  on  the  sequence  (or  order)  in  which  the 
information  is  presented  to  the  operators. 

Before  describing  the  experiment,  the  next  section  briefly  overviews  the 
Hogarth- Einhorn  model.  We  want  to  note  here  that  there  is  previous  empirical 
support  for  the  model  using  non-military  and  military  tasks.  For  example, 

Asare  (1990),  Ashton  and  Ashton  (1988,  1990),  and  Meisser  (1990)  present 
results  consistent  with  the  model's  predictions  using  professional  auditors. 
Serfaty,  Entin,  and  Tenney  (1989)  present  results  showing  an  order  effects 
bias  for  Amy  officers  perfoming  a  situation  assessment  task.  And  Adelman, 
Bresnick,  and  Tolcott  (1990)  found  results  consistent  with  the  model's 
predictions  for  an  Amy  air  defense  task  using  a  paper -and-pencil  instrument. 
The  current  experiment  examines  whether  these  results  would  be  replicated 
when  Army  air  defense  officers  use  the  actual  system. 

THEORETICAL  MODEL 

Theoretically,  the  Hogarth- Einhorn  model  predicts  that,  when  information 
is  presented  sequentially  and  a  probability  estimate  is  obtained  after  each 
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piece  of  information,  people  make  new  estimates  by  first  anchoring  on  the 
current  position,  and  then  adjusting  it  by  the  degree  to  which  the  new 
information  confirms  or  disconfirms  this  position.  Since  each  new  piece  of 
information  leads  to  a  new  anchor,  recent  information  is  weighted  more  than 
prior  information,  thereby  resulting  in  a  recency  affect  when  information  is 
presented  sequentially.  Moreover,  the  Hogarth- Einhorn  model  predicts  that  the 
larger  the  anchor,  the  greater  the  impact  of  the  same  piece  of  disconf irming 
information.  For  example,  the  higher  one's  probability  that  an  unidentified 
aircraft  is  a  friend,  the  greater  the  negative  impact  of  new  information 
indicating  that  it  is  a  foe.  Conversely,  the  smaller  the  anchor,  the  greater 
the  impact  of  the  same  confirming  information.  Consequently,  the  order  in 
which  the  same  confirming  and  disconf irming  information  is  sequentially 
presented  is  predicted  to  result  in  different  final  probability  estimates,  for 
information  order  is  predicted  to  significantly  affect  the  anchoring  posi¬ 
tions  . 

Figure  1  presents  a  notional  representation  of  the  model's  predictions 
for  the  air  defense  task.  We  assume  that  after  the  first  piece  of  informa¬ 
tion,  the  air  defense  operator  thinks  that  the  probability  that  the  unknown 
aircraft  is  a  friend  is  0.65.  For  Order  #1,  the  operator  first  receives 
confirming  information  further  suggesting  that  the  aircraft  is  a  friend,  and 
then  disconf irming  information  suggesting  that  it  is  a  foe.  For  Order  #2,  the 
same  information  is  presented  but  in  the  opposite  order.  Receiving  the  same 
information  in  different  ordered  sequences  is  quite  possible  in  the  air 
defense  task  depending  on  the  aircraft's  flight  path  and  actions,  and  the  way 
the  information  is  relayed  to  the  operator.  The  predictions  are  that  air 
defense  operators  will  give  a  significantly  higher  probability  of  friend  to 
Order  #2  than  to  Order  #1  simply  because  of  the  anchoring  and  adjustment 
heuristic  we  hypothesize  they  use  to  process  information. 

In  the  example  just  provided,  the  information  was  presented  sequentially 
to  the  operator.  This  is  the  principle  case  for  which  the  Ho garth -Einhorn 
model  predicts  that  an  anchoring  and  adjustment  heuristic  will  result  in  an 
order  effects  bias.  In  contrast,  when  information  is  presented  all  at  once, 
the  Hogarth- Einhorn  model  predicts  that  the  information  will  be  considered  in 
the  aggregate  or  "globally.”  Consequently,  the  order  in  which  confirming  and 
disconf irming  information  is  presented  should  have  no  effect  on  people's  final 
probability  estimates.  This  prediction  is  represented  by  the  circle  within  a 
circle  after  infoirmation  item  #3  in  Figure  1.  That  is,  operators  should  have 
the  same  final  probability  estimate  regardless  of  the  information's  order. 

THE  EXPERIMENT 


This  part  of  the  paper  is  divided  into  five  sections.  The  first  section 
describes  the  participants  and  research  site.  The  second  describes  the 
research  design.  The  third  section  describes  the  experimental  procedures. 

The  fourth  section  presents  the  hypotheses .  And  the  fifth  section  presents 
the  results. 
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Figure  1:  Notational  Representation  of  Model's  Predictions 
Research  Participants ,  Site,  and  Date 

The  experiment  was  conducted  during  the  week  of  29  October  -  2  November 
1990  with  twenty- four  officers  trained  as  Tactical  Control  Officers  for  the 
Patriot  Air  Defense  System.  The  task  was  implemented  using  the  Patriot 
training  simulators  in  the  Directorate  of  Training  Development  at  Ft.  Bliss, 
TX. 

Research  Design 

Independent  Variables .  Four  independent  variables  were  addressed  when 
constructing  the  experiment.  The  first  three  listed  below  were  used  to 
develop  the  aircraft  tracks  or  paths  that  the  aircraft  follows,  displayed  to 
the  participants ,  The  fourth  was  used  to  create  the  two  different  contexts 
under  which  the  tracks  were  displayed. 

The  first  independent  variable  was  whether  the  first  piece  of  information 
about  an  aircraft  indicated  that  it  was  friendly,  hostile,  or  unknown.  The 
second  independent  variable  was  the  order  in  which  confirming  and  disconfirm- 
ing  information  was  presented  early  in  an  aircraft's  track  history.  For  Early 
Order  #1,  the  second  item  of  information  confirmed  the  initial  information  and 
the  third  item  of  information  disconfirmed  it  (CD) .  The  opposite  order  was 
used  for  Early  Order  #2;  that  is,  disconfirm  then  confirm  (DC).  The  third 
independent  variable  was  the  order  in  which  confirming  and  disconf irming 
information  was  presented  late  in  an  aircraft's  track  history.  For  Late  Order 
#1 ,  the  fourth  item  of  information  confirmed  the  first  information  item,  and 
the  fifth  item  was  disconf irming  (CD).  For  Late  Order  #2,  the  opposite  order 
(DC)  was  used  for  the  fourth  and  fifth  information  items.  (Note:  For  tracks 
whose  first  piece  of  information  indicated  unknown,  the  confirming  information 
indicated  a  friendly  aircraft  and  the  disconf irming  information  indicated  a 
hostile. ) 
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Table  1  summarizes  the  research  design  for  the  three  independent  vari¬ 
ables  discussed  above.  The  twelve  cells  in  the  matrix  represent  the  twelve 
different  tracks  defined  by  crossing  the  three  independent  variables.  Each  of 
the  twelve  tracks  was  used  in  two  scenarios.  The  two  scenarios  were  used  to 
create  the  two  different  contexts  under  which  the  tracks  were  displayed. 


INITIAL 

INFORMATION 

EARLY  ORDER 

CONFIRM  (C) /DISCONFIRM  (D) 
LATE  ORDER 

DISCONFIRM  (D) /CONFIRM  (C) 
LATE  ORDER 

CD 

DC 

CD 

DC 

UNKNOWNS 

FRIENDS 

HOSTILES 

Table  1:  Research  Design  for  Each  Level  of  "Scenario”  Factor 

The  scenarios  represented  the  fourth  independent  variable.  In  the  first 
scenario,  there  was  only  one  aircraft  track  on  the  Patriot  display  at  a  time. 
Operators  received  information  sequentially  in  this  context,  for  there  was 
nothing  to  distract  them  from  observing  each  piece  of  information  as  it 
appeared  on  the  display.  In  the  second  scenario,  however,  there  were  from 
five  to  ten  aircraft  tracks  on  the  display  at  the  same  time.  In  this 
condition,  operators  have  too  many  tracks  to  monitor  each  one  continuously. 
Consequently,  we  reasoned  that  they  would  focus  on  tracks  only  when  the  tracks 
were  close  enough  to  be  possible  threats.  At  that  time,  they  would  call  up 
the  information  about  the  track  and,  thus,  receive  all  the  information  about 
it  at  once,  not  sequentially.  We  hypothesized  that  this  would  make  the  par¬ 
ticipants  use  a  global  information  processing  strategy  instead  of  an  anchoring 
and  adjustment  one. 

Two  steps  were  taken  to  minimize  participants'  ability  to  recognize 
previous  tracks  that  had  the  same  information  presented  in  a  slightly  differ¬ 
ent  order.  We  were  particularly  concerned  about  the  operators  recognizing 
previous  tracks  when  there  was  only  one  track  on  the  screen  at  a  time. 

First,  we  interspersed  six  other  tracks  among  the  twelve  comprising  the 
research  design.  These  six  tracks  had  no  conflicting  information.  Three  of 
the  tracks  were  "obvious  friends;"  all  information  indicated  that  the  aircraft 
was  friendly.  And  three  tracks  were  "obvious  hostiles;"  all  information 
indicated  that  the  aircraft  was  hostile. 

Second,  we  created  equivalent  variations  of  the  tracks.  This  was 
accomplished  in  a  two-step  procedure.  First,  we  created  "mirror  images"  of 
the  tracks.  For  example,  there  were  two  tracks  that  had  the  first  piece  of 
information  indicate  friend,  and  had  Early  Order  be  Confirm  then  Disconfirm 
(CD) .  Since  these  two  tracks  had  the  same  first  three  pieces  of  information. 
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we  had  one  of  the  tracks  appear  on  the  left-hand  side  of  the  Patriot  display 
and  the  other  appear  on  the  right-hand  side.  That  is,  the  first  half  of  one 
track  was  a  mirror  image  of  the  other.  They  had,  of  course,  different  Late 
Order  conditions.  Therefore,  in  the  second  step,  we  designed  the  flight  paths 
so  that  late  order  differences  were  not  affected  by  the  aircraft's  position  on 
the  display.  As  a  result,  the  tracks  looked  very  different.  However,  each 
set  of  four  friends,  four  foes,  and  four  unknowns  were  equivalent  except  for 
the  order  in  which  the  same  information  was  presented  to  the  operators. 

Dependent  Variables .  Three  dependent  variables  were  used  to  assess  if  the 
independent  variables  affected  the  participants'  judgments.  The  first  depen¬ 
dent  variable  was  the  probability  estimate  that  the  participants  gave  for 
whether  the  aircraft  was  friend  or  foe. 

The  second  was  the  participants'  actual  identification  of  the  aircraft  as 
friend,  foe  or  unknown.  For  example,  it  is  quite  possible  that  information 
order  significantly  affects  the  probabilities  given  to  two  friendly  aircraft 
but,  in  both  cases,  the  aircraft  are  identified  as  "friends."  The  identifi¬ 
cation  judgment,  not  the  probability  estimate,  is  the  more  important  measure 
of  air  defense  officers'  judgments  and  is  closer  to  the  task  they  actually 
perform. 

The  third  dependent  variable  was  the  participants'  decision  to  engage  an 
aircraft  or  not.  The  engagement  decision  is  highly  correlated  with,  but  not 
identical  to,  the  identification  decision.  The  engagement  decision  also  is 
affected  by  other  factors,  such  as  an  aircraft's  perceived  threat.  More 
importantly,  however,  it  is  the  ultimate  behavioral  measure  of  air  defense 
officers '  judgments . 

Experimental  Procedures 

The  experiment  was  implemented  on  the  Patriot  training  simulators  at  Ft. 
Bliss.  Each  session  had  three  or  four  participants.  The  simulation  system 
displayed  the  aircraft  tracks  to  the  participants  at  the  same  time.  However, 
participants  worked  independently  at  separate  Patriot  consoles,  and  were  free 
to  process  the  track  as  they  thought  fit. 

The  session  began  with  an  introduction  and  brief  overview  of  what 
probability  estimates  of  friend  or  foe  mean.  A  "Gulf  War”  situation  (prior  to 
the  Gulf  War)  was  presented  to  provide  a  context  for  the  session.  However, 
the  display  provided  no  unique  geographic  information.  The  description  of  the 
situation  justified  why  participants  were  working  in  an  autonomous  mode;  that 
is,  without  communications  with  other  units  or  headquarters.  By  being  in 
autonomous  mode,  the  participants  were  solely  responsible  for  their  identifi¬ 
cation  and  engagement  decisions. 

Participants  were  also  told  that  there  had  been  some,  but  minimal 
compromise  of  the  IFF  Mode  3  response  to  Patriot's  electronic  interrogation 
signal;  consequently,  this  response  was  not  a  perfect  indicator  of  friendly 
aircraft.  The  description  of  the  situation  also  indicated  that  it  was  equally 
likely  that  friendly  and  hostile  aircraft  were  observed  in  order  to  nullify 
participants  perceptions  of  this  prior  information.  Finally,  Patriot  was 
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operated  in  the  semi-automatic  engagement  mode  to  ensure  that  all  the  informa¬ 
tion  was  collected  for  all  participants.  However,  Patriot  continued  to  make 
identification  recommendations  based  on  its  interpretation  of  the  information. 

For  the  "one  track  at  a  time"  scenario,  the  track  display  and  console 
were  stopped  after  each  information  item  for  fifteen  seconds  for  data  collec¬ 
tion.  This  was  done  to  ensure  that  all  data  was  collected  at  the  same  point 
in  a  track's  history  for  all  participants.  The  data  was  collected  by  an 
observer  working  with  each  participant.  At  each  stopping  point,  the  partici¬ 
pant  (1)  told  the  observer  the  probability  that  the  aircraft  was  friendly  (or 
hostile);  (2)  identified  the  aircraft  as  friend,  hostile,  or  unknown;  and  (3) 
indicated  whether  or  not  they  would  engage  the  aircraft.  In  addition, 
participants  thought  aloud  throughout  the  session,  which  was  tape  recorded. 

If  at  any  time  in  a  track's  history  they  indicated  that  they  would  engage  the 
aircraft,  the  aircraft  was  considered  engaged.  However,  they  were  told  not  to 
actually  engage  the  aircraft  so  that  we  could  continue  to  collect  data  on  the 
two  other  dependent  variables. 

Figures  2  and  3  illustrate  how  this  procedure  worked  for  two  tracks. 
Figure  2  represents  one  of  the  "obvious  friends"  used  in  the  experiment.  The 
air  defense  sector  is  represented  by  the  inverted  pyramid.  The  operator  is 
located  at  the  apex  (bottom  of  figure)  and  protecting  two  assets.  The 
aircraft  first  appears  at  point  1.  The  console  is  frozen,  and  the  data 
collected,  at  point  A.  After  fifteen  seconds,  the  console  is  unfrozen  and  the 
track  continues  to  point  2  where  there  is  a  Mode  1  and  Mode  3  response  to  an 
automatic  IFF  interrogation  at  the  Forward  Support  Coordination  Line  (FSCL) . 
The  console  is  frozen  immediately  after  this  (friendly)  information  is 
presented,  and  the  data  is  collected.  After  fifteen  seconds  the  track  begins 
again.  It  goes  right  down  one  of  the  two  safe  passage  corridors,  just  as  a 
friendly  aircraft  should,  for  the  rest  of  its  flight  path.  The  console  is 
frozen,  and  data  collected,  at  points  C,  D,  E,  and  F. 

Figure  3  shows  one  of  the  twelve  tracks  in  the  research  design.  It  was 
conflicting  information,  the  track  first  appeared  at  the  FSCL,  and  the 
operator  receives  a  Mode  1  and  3  IFF  Response  indicating  friend.  Then,  the 
aircraft  proceeds  down  the  safe-passage  corridor;  this  is  confirming  informa¬ 
tion.  However,  at  point  2,  the  aircraft  begins  jamming  friendly  radar.  This 
is  disconfirming  information.  So,  the  Early  Order  is  Confirm/Disconfirm.  The 
aircraft  stops  jamming  at  point  3.  This  information  is  confirming  because  it 
suggests  friend.  Then,  however,  the  aircraft  goes  out  of  and  back  into  the 
safe  passage  corridor.  Even  though  the  aircraft  is  back  in  the  corridor  at 
point  E  where  the  fifth  set  of  data  is  collected  for  this  track,  the  "in- out 
move"  suggests  the  aircraft  is  hostile  and  therefore,  disconfirms  the  first 
piece  of  information.  Consequently,  the  Late  Order  is  also  Con¬ 
firm/Disconfirm. 

Figure  4  is  the  mirror  image  of  the  track  shown  in  Figure  3.  Its  flight 
path  is  the  same  except  it  is  coming  down  the  left-hand  side  of  the  display. 
The  Late  Order  condition  is,  of  course,  different  for  this  track.  The  "out- in 
move"  is  the  fourth  item  of  information  (i.e.,  disconfirming)  and  "stopped 
jamming"  is  the  fifth  information  item  (i.e.,  confirming).  Consequently,  its 
Late  Order  is  Disconf irm/Conf irm. 
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TRACK  18 


1 .  Track  appears  at  point  1 

2.  Auto  IFF  at  FSCL  at  point  2  (Mode  1 ,  Mode  3  response) 

3.  Follows  corridor  to  points  3  and  4 

1  • 

/- -  A 


Figure  2:  Aircraft  Track  for  a  Obvious  Friend 
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TRACK  8 


1 .  Track  appears  at  point  1  along  FSCL,  auto  IFF  (Mode  1 ,  Mode  3  response) 

2.  Track  follows  corridor  to  point  2,  starts  jamming 

3.  Stops  jamming  at  point  3 

4.  Leaves  corridor  at  point  4 

5.  Returns  to  corridor  at  point  5,  continues  in  corridor  to  point  6 


Figure  3:  Friendly  Track  with  Conflicting  Information 
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TRACK  16 

1 .  Track  appears  at  point  1  along  FSCL,  Auto  IFF  (Mode  1 ,  Mode  3  response) 

2.  Follows  corridor  to  point  2,  starts  jamming 

3.  Leaves  corridor  at  point  3,  returns  at  point  4,  stops  jamming  at  point  5 

4.  Continues  down  corridor  to  point  6 


FSCL 


asset! ^  [asset' 


Figure  4:  "Mirror  Image"  Friendly  Track 


The  displays  were  not  stopped  for  the  "many  track"  scenario  in  order  to 
maintain  time  pressure  on  the  participants.  Instead,  at  a  pre- determined 
point  for  each  of  the  twelve  tracks  comprising  the  research  design,  partici¬ 
pants  were  told  that  in  fifteen  seconds  they  would  have  to  tell  the  observer 
their  probability,  identification,  and  engagement  decision  for  the  track. 

This  point  was  determined  so  that  the  aircraft  would  be  at  point  E  for  the 
track.  Pilot- testing  indicated  that  this  was  enough  time  to  hook  the  track 
and  call-up  information  about  it.  No  two  design  tracks  were  on  the  screen  at 
the  same  time,  and  each  was  interspersed  with  5-10  filler  tracks.  The  "many 
track"  scenario  was  always  the  second  scenario  for  logistics  reasons. 

It  is  important  to  emphasize  that  (a)  the  two  scenarios  and  (b)  tracks 
with  conflicting  information  represent  situational  contexts  that  can  occur. 
When  the  U.S.S.  Vincennes  shot  down  the  Iranian  airliner,  there  were  only  two 
aircraft  tracks  on  its  display.  And,  during  the  Gulf  War,  there  was  often 
only  one  track  on  the  Patriot  display  at  a  time.  Although  these  tracks  repre¬ 
sented  high  speed  tactical  ballistic  missiles  that  were  automatically  engaged 
by  Patriot,  they  could,  of  course,  have  been  aircraft. 

More  routinely,  the  Army  air  defense  community  has  trained  for  the  "many 
track"  scenario  because  of  its  focus  on  Europe.  And  it  has  typically  trained 
for  tracks  without  conflicting  information.  However,  it  is  well  aware  that 
tracks  can  have  conflicting  information,  as  illustrated  by  the  Vincennes 
tragedy.  In  addition,  tracks  with  conflicting  information  sometimes  occur  in 
Europe . 

Hypotheses 

We  predicted  that  information  order  would  affect  all  three  measures  of 
the  operators'  judgments  only  when  the  information  was  presented  sequentially 
(i.e.,  for  the  "one  track  at  a  time"  scenario),  but  not  when  it  was  presented 
all  at  once  (i.e.,  for  the  "many  track"  scenario).  Consistent  with  the 
Hogarth- Einhorn  belief  updating  model,  we  predicted  this  would  occur  because 
operators  would  use  sequential  anchoring  and  adjustment  heuristic  when 
processing  information  in  the  former  case,  and  a  global  information  processing 
strategy  in  the  latter.  Only  the  sequential  anchoring  and  adjustment  heuris¬ 
tic  was  predicted  to  result  in  an  information  order  bias. 

Results 

Before  examining  the  data  for  the  tracks  with  conflicting  information, 
which  comprised  the  research  design,  we  performed  two  manipulation  checks. 
First,  we  examined  the  data  for  the  tracks  without  conflicting  information 
(i.e.,  the  "obvious  friends"  and  "obvious  hostiles")  to  determine  if  the 
operators  treated  them  as  such.  The  data  indicated  that  they  did.  Specifi¬ 
cally,  the  mean  probability  of  friend  estimate  for  the  three  "obvious 
friends"  at  Point  E  was  0.91.  On  the  average,  22  of  the  24  participants  had 
identified  them  as  friends  by  Point  E,  and  there  were  no  engagements.  In 
contrast,  the  mean  probability  of  friend  estimate  for  the  three  "obvious 
hostiles"  was  0.12  at  Point  E.  On  the  average,  21.33  of  the  participants  had 
made  a  "hostile"  identification  by  Point  E  and,  on  the  average,  there  were 
17.33  engagements  by  Point  E. 
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The  second  manipulation  check  focused  on  the  twelve  tracks  comprising  the 
research  design.  We  wanted  to  make  sure  that  there  was  only  one  new  piece  of 
information  about  a  track  each  time  the  display  was  frozen  for  data  collection 
purposes  in  the  "one  track  at  a  time  scenario".  If  two  pieces  of  information 
were  presented  in  between  data  collection  stops,  then  it  would  be  impossible 
to  assess  the  effect  of  each  information  item.  More  importantly,  if  two 
pieces  of  information  were  presented  together  then,  according  to  the  Hogarth- 
Einhorn  belief  updating  model,  the  operators  would  use  a  global  processing 
strategy  instead  of  a  sequential  anchoring  and  adjustment  heuristic  for  the 
"one  track  at  a  time"  scenario.  Consequently,  we  would  be  unable  to  ascertain 
whether  a  failure  to  find  the  hypothesized  "information  order  bias"  would  be 
due  to  an  experimental  manipulation  failure  or  the  failure  to  reject  the  null 
hypothesis,  which  was  that  information  order  would  have  no  affect  on  air 
defense  officers'  judgments. 

The  second  manipulation  check  was  of  particular  concern  because  pilot 
testing  had  indicated  that  it  was  extremely  difficult  to  construct  the  tracks 
so  that  two  pieces  of  information  did  not  appear  together.  This  difficulty 
occurred  because  the  training  simulators  were  not  designed  to  have  the 
precision  required  to  construct  the  kinds  of  tracks  needed  for  the  experiment. 
There  was  only  one  month  between  the  pilot  test  and  the  dates  for  conducting 
the  experiment.  These  dates  were,  so  to  speak,  fixed  in  stone.  Corrections 
were  being  made  on  the  tracks  right  up  to  the  day  before  the  experiment  began. 

Careful  examination  of  hard-copies  of  the  displays  after  the  experiment 
was  completed  indicated  that  a  number  of  cells  in  the  design  were  not  imple¬ 
mented  perfectly.  Despite  all  the  efforts  of  participating  research  personnel, 
sometimes  two  pieces  of  information  (instead  of  one)  appeared  on  the  Patriot 
display  at  critical  times  prior  to  obtaining  participants'  judgments  in  the 
"one  track  at  a  time"  scenario.  The  unshaded  portion  of  the  boxes  in  Table  2 
indicate  which  half  of  a  track's  history  was  implemented  perfectly. 


INITIAL 

INFORMATION 

EARLY  ORDER 

CONFIRM  (C)/DISCONFIRM  (D) 
LATE  ORDER 

DISCONFIRM  (D) /CONFIRM  (C) 
LATE  ORDER 

CD 

DC 

CD 

DC 

UNKNOWNS 

IHHI 

FRIENDS 

HOSTILES 

Table  2:  Unshaded  Boxes  Indicate  Which  Half  of  a  Track's  History  was 

Implemented  Perfectly 


The  rest  of  this  section  presents  the  results  of  the  analysis  for  (1)  the 
early  order  manipulation  for  the  two  Unknown  tracks  that  were  implemented 
perfectly,  (2)  the  Late  Order  manipulation  for  the  three  Unknown  tracks  whose 
late  order  manipulation  was  implemented  perfectly.  Each  analysis  is  present¬ 
ed,  in  turn. 

Early  Order:  Unknown  Tracks.  We  began  by  first  calculating  the  differ¬ 
ence  score  for  each  participant's  probability  estimates  between  Points  C  and  A 
for  the  two  Unknown  tracks.  Difference  scores  were  used  to  control  for 
differences  in  participants'  starting  probability  estimates  at  point  A.  Then, 
we  performed  a  within- subject  Analysis  of  Variance  (ANOVA)  using  the  SAS  soft¬ 
ware  package  for  Personal  Computers  to  assess  whether  there  was  a  significant 
mean  difference  in  the  difference  scores  for  (1)  the  Unknown  track  with  a  CD 
early  order,  and  (2)  the  Unknown  track  with  a  DC  early  order.  There  was  not; 
the  mean  difference  score  for  each  track  was  essentially  zero.  This  indicates 
that,  for  the  one  test  performed,  the  order  of  information  early  in  a  track's 
history  had  no  effect  on  operators'  judgments  for  the  "one  track  at  a  time" 
scenario . 

Late  Order:  Unknown  Tracks.  We  calculated  the  difference  score  for  each 
participant's  probability  estimates  between  Points  E  and  C  for  the  three 
Unknown  tracks  that  were  implemented  perfectly  for  the  "one  track  at  a  time" 
scenario . 

Difference  scores  were  used  to  control  for  differences  in  participants' 
starting  probability  estimates  at  point  C.  The  difference  scores  were  not 
significantly  different. 

Late  Order:  Friendly  Tracks.  We  calculated  the  difference  score  for 
each  participant's  probability  estimates  between  Points  E  and  C  for  the  two 
Friendly  tracks  that  were  implemented  perfectly  for  the  "one  track  at  a  time" 
scenario.  These  two  tracks,  referred  to  as  Tracks  #8  and  #16,  are  the  respec¬ 
tive  tracks  shown  previously  in  Figures  3  and  4.  Both  tracks  have  a  Con- 
firm/Disconfirm  (CD)  early  order,  and  Track  #16  has  a  Disconf irm/Conf irm  (DC) 
late  order. 

Consistent  with  our  hypothesis,  Track  #8  had  a  lower  mean  difference 
score  than  Track  #16;  -0.10  versus  +0.03.  A  within- subject  ANOVA  found  this 
difference  significant  at  the  p  -  0.05  level.  This  finding  supports  the 
predictions  of  the  Hogarth- E inhorn  model,  and  suggests  that  the  participating 
operators  were  using  an  anchoring  and  adjustment  heuristic  to  process  informa¬ 
tion  in  the  "one  track  at  a  time"  scenario. 

We  then  added  Track  #7  to  the  analysis.  The  early  order  manipulation  for 
Track  #7  was  not  implemented  perfectly,  but  the  late  order  manipulation  was. 
The  late  order  manipulation  for  Track  #7  was  Disconf irm/Conf irm.  A  within- 
subjects  ANOVA  found  the  E-C  difference  score  for  Tracks  #7  and  #8  to  be 
significantly  different  at  the  p  «  0.02  level.  These  results  replicate  those 
for  Tracks  #8  and  #16 .  The  mean  difference  scores  for  Tracks  #7 ,  #8 ,  and  #16 
are  presented  in  Figure  5. 
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Confirm/Disconfirm  Disconfirm/Confirm 


{Points  D  and  E) 


Figure  5:  Mean  Differences  for  Friendly  Tracks  #7,  #8,  #16 

The  above  analysis  indicates  that  the  order  of  information  late  in  the 
history  of  Tracks  #7,  #8,  and  #16  did  affect  operators'  probability  estimates. 
We  now  address  whether  it  affected  the  operators'  identification  judgments. 

We  could  not  perform  a  chi -squared  test  because  each  participant  made  an  iden¬ 
tification  for  each  track;  consequently,  the  entries  in  a  chi -squared  table 
would  not  be  independent.  However,  we  could  use  all  the  information  to  perform 
a  test  of  marginal  homogeneity.  This  test  assessed  whether  the  distribution 
of  identifications  was  significantly  different  for  the  three  tracks. 

Table  3  presents  the  identification  matrix  for  Tracks  #7,  #8,  and  #16. 
This  is  the  matrix  used  in  the  test  of  marginal  homogeneity.  Although  Track 
#8  has  three  more  "hostile"  identifications  than  either  Track  #7  or  #16,  the 
distribution  of  identifications  look  quite  similar  for  the  three  tracks.  This 
observation  was  supported  by  the  marginal  homogeneity  test,  which  found  that 
the  distributions  were  not  significantly  different. 


IDENTIFICATION 

HOSTILE 

UNKNOWN 

FRIEND 

TRACK  #8  (CD) 

7 

7 

10 

TRACK  #16  (DC) 

4 

7 

13 

TRACK  #7  (DC) 

4 

10 

10 

X]  =  2.65,  P  =  .62 


Table  3:  Identification  Matrix  for  Tracks  #7,  #8,  and  #16  at  Point  E 
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Next,  we  examined  whether  there  was  a  difference  in  the  distribution  of 
engagement  decisions  for  the  three  tracks.  There  were  no  engagements  for  any 
of  the  three  tracks  at  Point  C.  Table  4  shows  the  number  of  engagements  for 
each  track  at  Point  E.  As  would  be  predicted  by  the  Hogarth- Einhorn  model, 
there  were  more  engagements  for  Track  #8  than  Tracks  #7  and  #16.  Moreover, 
the  test  of  marginal  homogeneity  found  this  distribution  to  be  significant  at 
the  p  =  0.01  level.  Not  only  did  the  Late  Order  manipulation  affect  operato¬ 
rs'  probability  estimates  for  these  three  tracks,  it  affected  their  engagement 
decisions  as  well. 


ENGAGEMENT 

NO 

YES 

TRACK  #8  (CD) 

15 

9 

TRACK  #16  (DC) 

19 

5 

TRACK  #7  (DC) 

21 

3 

-  8.73,  P  -  .013 


Table  4:  Engagement  Matrix  for  Tracks  #7,  #8,  and  #16  at  Point  E 

But  why  was  there  a  significant  difference  in  the  distribution  of 
engagements  and  not  the  identifications?  Examination  of  the  data  indicates 
that  all  seven  participants  who  identified  Track  #8  as  "hostile,”  engaged  it. 
In  addition,  two  of  the  participants  who  identified  Track  #8  as  "friendly"  at 
Point  E,  engaged  it  between  Points  D  and  E.  That  is,  they  engaged  the  track 
when  it  began  the  "out-in  move"  shown  in  Figure  3.  The  "out-in-move"  is 
disconfirming  information.  It  is  critical  information  suggesting  a  hostile 
aircraft,  and  it  occurred  late  in  the  track's  history.  Even  though  both 
participants  later  said  the  aircraft  was  a  friend  after  it  had  returned  to  the 
safe -passage  corridor,  they  had  long  since  shct  it  down. 

The  final  analysis  focused  on  the  "all  at  once"  scenario.  Consistent 
with  the  Hogarth- Einhorn  model,  the  type  of  scenario  did  affect  the  results. 
Table  5  presents  the  mean  probability  of  friend  estimates  for  Tracks  #7,  #8, 
and  #16  at  Points  E  and  F  for  both  scenarios.  As  can  be  seen,  the  mean  esti¬ 
mate  for  Track  #8  only  appears  significantly  lower  for  the  "one  track  at  a 
time"  scenario  (i.e..  Scenario  #1).  As  predicted,  the  mean  probability 
estimates  for  the  three  tracks  are  about  the  same  for  the  "all  at  once" 
scenario  (i.e.,  Scenario  #2).  The  interaction  for  the  2  Scenario  by  3  Tracks 
within- subjects  ANOVA  approached  significance  (p  -  0.09)  at  Point  E,  and  was 
significant  (p  -  0.03)  at  Point  F. 
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Mean  Probabilities  at 

Point  E 

Track  #7 

Track  #8 

Track  #16 

"One  at  a  Time"  Scenario 
(Scenario  1) 

.65 

.51 

.66 

"All  at  Once"  Scenario 
(Scenario  2) 

.63 

.62 

.67 

Mean  Probabilities  at  Point  F 
Track  #7  \  Track  #8  I  Track  #16 


"One  at  a  Time"  Scenario 
(Scenario  1) 

.75 

.58 

.69 

"All  at  Once"  Scenario 
(Scenario  2) 

.72 

.69 

.72 

Table  5:  Mean  Probability  of  Friend  Estimates  for  Tracks  #7,  #8,  and  #16  for 

Both  Scenarios 


The  type  of  scenario  did  not  affect  the  identification  decision  for  Track 
#8  at  Point  E.  However,  it  did  affect  the  engagement  decision.  Table  6  shows 
the  engagement  decisions  the  24  air  defense  officers  made  for  Track  #8  in  the 
two  scenarios.  As  can  be  seen,  eight  of  the  nine  participants  who  engaged 
Track  #8  in  the  "one  track  at  a  time"  scenario  did  not  engage  it  in  the  "all 
at  once"  scenario.  The  test  of  marginal  homogeneity  was  significant  at  the  p  » 
0.04  level,  indicating  that  the  distribution  of  engagements  for  Track  #8  was 
significantly  different  for  the  two  scenarios.  In  contrast,  the  distribution 
of  engagements  was  not  significantly  different  for  Tracks  #7  or  #16. 


Table  6:  The  Engagement  Decision  Made  for  Track  #8  in  Both  Scenarios  at  Point  E 


DISCUSSION 


The  experiment  reported  herein  obtained  results  consistent  with  the 
predictions  of  the  Ho gar th-E inhorn  belief  updating  model  for  the  late  order 
manipulation  for  three  Friendly  tracks  (#7,  #8,  and  #16).  When  information 
was  presented  sequentially  in  the  "one  track  at  a  time"  scenario,  and  a 
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probability  estimate  was  obtained  after  each  piece  of  information,  the 
Confirm/Disconf irm  late  order  manipulation  resulted  in  a  significantly  lower 
mean  probability  of  friend  estimate,  and  a  significantly  larger  number  of  en¬ 
gagements,  than  the  Disconf irm/Conf irm  late  order  manipulation.  These 
differences  were  not  found  when  information  was  presented  all  at  once.  These 
results  support  the  hypothesis  that  air  defense  officers  use  an  anchoring  and 
adjustment  heuristic  when  information  is  presented  sequentially,  and  a  more 
global  aggregation  strategy,  such  as  a  weighted  averaging  strategy,  when 
information  is  presented  all  at  once.  Finally,  it  is  important  to  emphasize 
that  these  results  were  obtained  when  information  was  presented  on  the  Patriot 
display,  and  that  they  replicate  previous  findings  obtained  with  a  paper-and- 
pencil  instrument  (Adelman,  Bresnick,  and  Tolcott,  1990). 

These  results  must,  however,  be  tempered  by  three  caveats.  First,  many 
of  the  cells  in  the  design  were  not  implemented  perfectly.  In  particular, 
there  was  only  one  Friendly  track  (Track  #8)  whose  Confirm/Disconf  irm  late 
order  manipulation  was  implemented  perfectly.  Second,  there  was  no  "late 
order"  effect  for  the  three  Unknown  tracks  whose  late  order  manipulation  was 
implemented  perfectly.  And,  third,  there  was  no  "early  order"  effect  for  the 
two  perfectly  implemented  Unknown  tracks.  Consequently,  one  must  be  cautious 
in  our  conclusions  about  the  generality  of  the  observed  "information  order 
bias."  Further  research  is  needed  to  address  these  caveats.  Nevertheless, 
the  experiment  indicates  that  such  a  bias  can  occur  with  highly  trained 
military  personnel,  and  that  it  can  result  in  serious  consequences;  in 
particular,  an  increase  in  the  probability  of  engaging  a  friendly  aircraft. 

Actions  in  the  recent  Gulf  War  have  clearly  demonstrated  that  Patriot  air 
defense  operators  performed  extremely  well,  and  that  the  Patriot  air  defense 
system  is  an  excellently  engineered  weapon  system.  Our  findings  do  not 
detract  from  these  facts.  However,  they  do  suggest  that  these  high  perfor¬ 
mance  levels  might  be  even  further  enhanced  by  considering  how  operators 
process  information.  Future  research  is  directed  toward  (1)  investigating  the 
generality  of  the  observed  order  effects,  and  (2)  whether  display  modifica¬ 
tions  designed  to  facilitate  a  global  processing  strategy  can  remove  the 
information  order  bias.  More  generally,  the  experiment  report  herein  demon¬ 
strates  the  applied  implications  of  basic  research  investigating  human 
information  processing,  and  the  importance  of  understanding  cognitive  process¬ 
es  when  developing  computer  systems. 
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